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LONG-TERM  GOALS 

The  primary  goal  of  this  work  is  to  evaluate,  describe,  and  improve  the  performance  of  COAMPS®1 
cloud  forecasts  in  terms  of  their  ability  to  predict  clear  skies  for  airborne  intelligence  and 
reconnaissance  operations. 

OBJECTIVES 

The  objective  of  this  work  is  to  quantitatively  verify  the  quality  of  COAMPS®  clear-sky  forecasts  in 
terms  of  the  position  and  timing  of  clear-sky  events  primarily  over  the  Pacific  Ocean.  The  results  will 
be  communicated  in  terms  of  the  systematic  biases  as  well  as  the  ability  to  predict  clear  periods  over  a 
range  of  spatial  scales.  The  feasibility  of  applying  statistical  forecasts  and/or  physics-based 
corrections  will  also  be  investigated. 

APPROACH 

Clear  skies  typically  occur  either  as  irregularly  shaped  areas  between  cloudy  regions  or  as  wide  zones 
during  quiescent  periods.  The  complexity  of  the  cloud  patterns  requires  a  broad  range  of  measures  to 
fully  evaluate  the  numerical  forecasts  against  the  satellite  cloud  observations.  User  needs  also  play  a 
critical  role  in  determining  forecast  performance.  Statistics  such  as  the  bias  and  spatial/temporal  (lag) 
correlations  provide  basic  information  regarding  cloud  timing,  occurrence,  and  relative  frequency. 
More  sophisticated  methods,  such  as  spatial  compositing  and  neighborhood  averaging  provide 
information  regarding  structural  errors  in  the  cloud  field. 

To  provide  customized  statistics,  the  Center  for  Naval  Analysis  (CNA)  was  contacted.  Based  on 
guidance  provided  by  David  Ruskin,  verification  was  conducted  on  CO  AMPS®  forecasts  in  the 
Western  Pacific  centered  near  Taiwan  (Fig.  1).  Horizontal  grid  spacing  ranged  from  45  km  on  the 
outer  grid  to  5  km  on  the  inner  grids.  Convective  storms  were  explicitly  resolved  on  the  5  km  grids; 
the  Kain-Fritsch  convective  parameterization  scheme  was  used  to  represent  convective  effects  on  the 


1  CO  AMPS®,  the  Coupled  Ocean/ Atmosphere  Mesoscale  Prediction  System,  is  a  registered  trademark  of  the  Naval  Research  Laboratory. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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outer  grids.  Observational  cloud  data  consisted  of  liquid  water  path  and  cloud  cover  retrievals  from 
the  GOES  satellites,  as  well  as  CloudSat  and  MODIS  measurements.  The  observations  were  quality 
controlled  and  adjusted  for  biases  and  measurement  error.  Clear  sky  forecasts  were  validated  based  on 
liquid  water  path  thresholds  relevant  to  CNA  operations. 

Jason  Nachamkin  (NRL  code  7533)  was  responsible  for  the  verification  work.  CO  AMPS®  forecasts 
were  supplied  by  FNMOC2  as  well  as  by  John  Cook  and  Dan  Geiszler  (NRL  code  7542).  Satellite 
cloud  observations  were  provided  by  Kim  Richardson  (NRL  code  7541) 
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Figure  1.  The  COAMPS0  grid  configuration  for  the  western  Pacific  forecast  study.  The  45  km 
parent  grid  is  represented  by  the  mapped  area.  Nests  2  (yellow)  and  3  (orange)  with  spacings  of  15 
and  5  km,  respecitvely  are  centered  over  Taiwan.  Nests  4  (red)  and  5  (purple)  centered  over  Japan 

were  not  evaluated  in  this  study. 


WORK  COMPLETED 

An  in-depth  verification  study  of  the  western  Pacific  COAMPS®  clear-sky  forecasts  was  conducted. 
Approximately  one  year  of  24-hour  forecasts  spanning  March  2008  to  February  2009  were  evaluated. 
As  part  of  the  evaluation,  the  Neighborhood  verification  method  (Roberts  and  Lean  2008)  was 
implemented  to  perform  probabilistic  verification  of  the  spatial  distribution  of  clear  skies.  Results 


2  FNMOC  is  the  Fleet  Numerical  Meteorology  and  Oceanography  Center,  the  central-site  production  center  for  the  Navy’s  many  weather  products  that  are 
derived  from  automated  computer-based  data  assimilation  and  modeling  systems  such  as  NOGAPS  and  CO  AMPS  that  are  run  on  FNMOC’ s  high- 
performance  computers  . 
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(below)  indicate  a  significant  bias  exists  in  the  upper  tropospheric  cirrus  cloud  cover.  A  series  of 
sensitivity  studies  was  conducted  to  investigate  the  source  of  the  problem.  These  studies  revealed  that 
the  Kain-Fritsch  scheme  was  producing  an  overabundance  of  moisture  in  the  upper  atmosphere. 
Current  work  is  focused  on  adjusting  the  moisture  feedback  in  the  Kain-Fritsch  software  to  reduce  the 
amount  of  cloud  cover. 

RESULTS 

The  western  Pacific  forecasts  were  focused  primarily  in  the  subtropics  (Fig.  1),  where  relatively 
weakly  forced  convection  dominates.  Atmospheric  predictability  is  generally  lower  in  this  regime, 
and  as  a  result  the  forecast  accuracy  was  reduced  in  comparison  to  the  mid-latitude  eastern  Pacific 
forecasts  verified  previously.  Point-to-point  correlations  between  the  predicted  and  observed  LWP 
fields  were  on  the  order  of  0.3,  while  the  eastern  Pacific  values  were  about  0.5.  As  noted  by 
Nachamkin  et  al  (2009),  point-to-point  correlations  are  generally  lowered  by  small-scale  variability  in 
the  cloud  field,  and  subtropical  convection  generally  contributes  to  enhanced  variability. 

To  determine  the  basic  properties  of  the  cloud  distribution,  the  COAMPS  cloud  top  heights  were 
verified  against  the  satellite  retrievals.  Forecast  clouds  were  defined  as  the  highest  level  with 
measurable  liquid  or  ice  content,  while  the  satellite  retrieved  cloud-top  height  was  estimated  by 
converting  the  black-body  temperature  to  a  representative  atmospheric  height.  High  cloud  cover  was 
underestimated  by  the  satellite  due  to  upwelling  radiation  in  areas  of  thin  cirrus.  However  the  positive 
bias  in  the  forecast  cloudiness  was  still  significant  due  to  its  large  magnitude  (Fig.  2).  The  bias 
appeared  on  all  grids  and  all  forecast  lead  times,  and  was  most  intense  on  the  high  resolution  grids.  As 
mentioned  above,  the  Kain-Fritsch  parameterization  was  likely  contributing  a  major  portion  of  the 
additional  moisture.  Additional  tests  indicate  that  the  ice  nucleation  scheme  may  have  been 
contributing  to  the  additional  moisture  on  the  finest  (convective  resolving)  grid.  Both  issues  are 
currently  being  investigated. 


Figure  2.  Average  Cloud  top  height  profiles  for  the  5  and  1 7-hour  forecasts. 
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A  major  part  of  forecast  verification  is  determining  the  added  value  for  the  users.  In  the  current 
application,  the  users  were  interested  in  locating  clear  skies  within  specified  target  zones.  Perfect 
forecasts  were  not  necessary  as  long  as  clear  skies  occurred  within  an  acceptable  radius  of  the  initial 
target.  To  determine  the  viability  of  the  forecasts  in  this  situation,  the  Neighborhood  verification 
method  developed  by  Roberts  and  Lean  (2008)  was  implemented.  The  Neighborhood  method 
evaluates  forecast  reliability  at  successively  increasing  scales,  starting  at  a  single  grid  cell  (Fig.  3).  As 
the  size  of  the  evaluation  region  increases,  the  encompassed  forecast  and  observed  cloud  fractions  are 
less  sensitive  to  small  spatial  errors.  In  the  example  provided  in  Figure  3,  the  forecast  valid  at  the 
single  grid  cell  at  the  center  of  the  grid  is  incorrect  because  clear  skies  are  predicted  where  clouds  were 
observed.  However,  the  larger  bold  box  encompassing  the  5x5  grid  domain  contains  the  same  number 
of  cloudy  grid  cells  in  both  fields.  If  a  user  is  willing  to  accept  the  decreased  precision  incurred  from 
the  larger  region,  they  would  be  able  to  benefit  from  the  increased  reliability.  The  optimal  scale 
depends  on  the  cost  of  covering  large  areas  compared  to  the  benefits  of  completing  the  mission. 
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Figure  3.  Schematic  representation  of  the  Neighborhood  verification  method  as  depicted  in  Roberts 
and  Lean  (2008).  Gridded  observations  and  forecast  values  are  represented  by  the  small  grid 
squares.  Shaded  cells  represent  cloudy  grid  boxes.  Bold  boxes  represent  averaging  areas.  The 
numbers  beneath  each  grid  represent  the  cloud  fraction  within  the  large  and  small  bold  boxes. 


Reliability  diagrams  generated  by  applying  the  Neighborhood  method  to  the  western  Pacific  cloud 
forecasts  were  derived  to  depict  the  degree  of  the  match  between  the  observed  and  predicted  cloud 
fractions  (Fig.  4).  Forecasts  lying  on  the  45-degree  line  contain  the  same  cloud  fraction  as  the 
observations  and  are  thus  perfectly  reliable.  The  dashed  lines  depict  the  average  observed  cloud 
fraction  over  the  one-year  period.  Forecasts  lying  along  these  lines  are  essentially  random  and  are  of 
little  use.  The  results  from  the  45-km  grid  (Fig.  4)  indicate  that  the  reliability  of  the  CO  AMPS® 
forecasts  is  relatively  low  at  the  at  the  grid  scale,  but  steadily  increases  with  increasing  scale.  Due  to 
the  cloud  bias,  cloudy  forecasts  are  overabundant  and  the  greatest  improvements  occur  at  high  clear 
fractions.  In  general,  cloud  fraction  forecasts  over  regions  greater  than  495  km  (11  Ax)  retain 
appreciable  skill,  especially  when  clear  skies  are  predicted.  These  results  will  likely  improve 
considerably  once  the  cloud  bias  is  removed. 
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Figure  4.  Reliability  diagram  representing  the  performance  of  the  forecasts  on  the  4  5 -km  grid  at 
varying  scales.  Scale  is  defined  by  the  width  of  the  square  averaging  region  used  in  the 
Neighborhood  method.  The  Brier  score  represents  a  summary  performance  measure  of  forecast 
reliability.  Low  Brier  scores  are  desirable,  zero  represents  perfect  reliability. 


IMPACT/APPLICATIONS 

Prior  to  this  project,  the  systematic  statistical  performance  of  the  CO  AMPS®  clear  sky  and  cloud 
forecasts  was  largely  unknown.  Cloud  prediction  is  important  to  a  number  of  military  surveillance  and 
aviation  applications.  These  statistics  will  help  both  users  and  modelers  to  understand  and  ultimately 
improve  the  forecasts  for  future  use. 

TRANSITIONS 

The  verification  statistics  were  delivered  to  Andrew  Slaterbeck  (ONR).  The  results  will  be  used  by  the 
Center  for  Naval  Analysis  (CNA)  to  determine  the  viability  of  the  mesoscale  cloud  forecasts.  The 
Neighborhood  method  was  implemented  as  part  of  the  NRL  verification  package. 

RELATED  PROJECTS 

This  project  is  closely  related  to  the  cloud- liquid  and  ice-water  forecast  verification  and  improvement 
efforts  sponsored  by  the  Naval  Surface  Warfare  Center  (NSWC)  and  headed  by  Jerome  Schmidt  (NRL 
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code  7533).  Many  of  the  statistical  methods  to  compare  the  cloud  forecasts  with  the  observations  were 
developed  in  conjunction  with  this  project.  The  eastern  Pacific  satellite  cloud  data  were  also  collected 
under  this  project. 
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